[AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] by wjxiz1992 · Pull Request #15835 · NVIDIA/cudf-spark

wjxiz1992 · 2026-08-31T08:45:28Z

JaCoCo production line coverage: independently measured, not additive across incompatible shim bytecode tuples (sql-plugin +60, Spark 3.3/Scala 2.12; sql-plugin +48, Spark 4.0/Scala 2.13; measured head 4896e367)

Fixes #15511.

Description

GPU file scans can surface a raw FileNotFoundException after a cached data file is removed. Before Spark 4, this omits Spark's recovery guidance. On Spark 4.x, it also loses structured-error parity with CPU execution: the GPU path should preserve Spark's error condition and the exact missing PATH parameter.

This change handles both generations directly in the existing fix:

Spark 3.x retains the V1 REFRESH TABLE/recreate guidance and the V2 recreate-only guidance.
Spark 4.x uses Spark's structured missing-file error, preserving the CPU error condition and exact missing path for both V1 and V2 scans.
Direct and direct ExecutionException-wrapped missing-file failures are converted; unrelated exceptions and deeper wrapper chains are left unchanged.
Missing paths are attached where the individual file is known, including coalescing and multithreaded reader boundaries, then preserved through GpuDataSourceRDD and V2 reader factories.

The three inherited Spark tests excluded by #15511 are re-enabled, and focused RAPIDS coverage exercises the relevant paths:

RapidsMetadataCacheV1Suite: SPARK-16336,SPARK-27961 Suggest fixing FileNotFoundException maps to Spark MetadataCacheSuite.scala lines 41-63.
RapidsMetadataCacheV1Suite: SPARK-16337 temporary view refresh maps to Spark MetadataCacheSuite.scala lines 72-99.
RapidsMetadataCacheV2Suite: SPARK-16336,SPARK-27961 Suggest fixing FileNotFoundException maps to Spark MetadataCacheSuite.scala lines 41-63.
Spark 3.x missing ORC file includes recovery guidance - COALESCING/MULTITHREADED verifies the V1/V2-specific guidance.
Spark 4.x MissingFileStructuredErrorSuite covers V1 and V2 with COALESCING and MULTITHREADED, plus V1 PERFILE and the copied GpuFileScanRDD, comparing the CPU/GPU error condition and missing-path parameter and verifying GPU scan-plan evidence.
FileSystemBytesReadTrackerSuite: GPU datasource RDD enriches next() missing-file failures - direct V2/wrapped V1 directly covers reader failures and the metrics finally path.

Test	Tier	Action	Maven result	Status
V1 SPARK-16336/SPARK-27961 missing-file guidance	T1	Fix production exception guidance and remove exclusion	PASSED	RECOVERED
V1 SPARK-16337 temporary view refresh	T1	Add V1 refresh guidance and remove exclusion	PASSED	RECOVERED
V2 SPARK-16336/SPARK-27961 missing-file guidance	T1	Add V2 recreate guidance and remove exclusion	PASSED	RECOVERED
Spark 4.x V1/V2 structured-error parity	T1	Preserve Spark condition and exact missing path	PASSED	COVERED

Original Spark source: https://github.com/apache/spark/blob/f74867bddfbcdd4d08076db36851e88b15e66556/sql/core/src/test/scala/org/apache/spark/sql/MetadataCacheSuite.scala#L41-L99

Local validation:

Spark 3.3 focused suites plus GpuMultiFileReaderSuite: Tests: succeeded 17, failed 0, canceled 0, ignored 0, pending 0; All tests passed; BUILD SUCCESS.
Spark 4.0 focused GpuMultiFileReaderSuite and MissingFileStructuredErrorSuite, ANSI off/on: each run reported Tests: succeeded 8, failed 0, canceled 0, ignored 0, pending 0; BUILD SUCCESS.
Shim coverage check: No Origin.context leaks in shared source; shim signature coverage consistent across peers.
Spark 3.4 sql-plugin compile: BUILD SUCCESS.
Spark 4.2 sql-plugin compile: BUILD SUCCESS.
Spark 4.0 Databricks 17.3 validation could not reach affected-source compilation because spark-parent_2.13:4.0.0-databricks-173 is unavailable locally.
Spark 5.0 validation could not reach final affected-source compilation because rapids-4-spark-private_2.13:spark500:26.10.0-SNAPSHOT is unavailable locally.
JaCoCo production fix-line intersection at 4896e367: sql-plugin +60 of 162 added production lines on Spark 3.3/Scala 2.12, and sql-plugin +48 of 162 on Spark 4.0/Scala 2.13. These measurements use incompatible classfiles and are intentionally reported independently rather than summed.

Performance impact: a current-code local reader-wrapper microbenchmark measured a 0.695 ns/row baseline median and 0.698 ns/row wrapped median, a +0.003 ns/row (+0.45%) delta. The successful Spark 4/5 path uses direct try/catch boundaries around createReader, createColumnarReader, next, and get without the previous call-by-name helper; it adds no filesystem access, metadata reads, or GPU operations.

AI assistance: The change and PR description were prepared with Codex assistance and reviewed by the author before submission.

Checklists

Documentation

Updated for new or modified user-facing features or behaviors
No user-facing change

Testing

Added or modified tests to cover new code paths
Covered by existing tests
(Please provide the names of the existing tests in the PR description.)
Not required

Performance

Tests ran and results are added in the PR description
Issue filed with a link in the PR description
Not required

Signed-off-by: Allen Xu <allxu@nvidia.com>

greptile-apps · 2026-08-31T09:18:16Z

Greptile Summary

The PR restores Spark-compatible missing-file recovery behavior in GPU file scans and preserves missing paths across asynchronous and data-source reader boundaries. The latest changes retain the concrete reader-factory identity for reader-type inspection while applying structured missing-file conversion at the execution boundary.

Adds version-specific recovery guidance and structured Spark 4.x missing-file errors.
Propagates exact file paths through coalescing, multithreaded, V1, and V2 scan paths.
Restores inherited metadata-cache tests and adds focused coverage for reader modes, wrapper handling, metrics cleanup, and CPU/GPU error parity.
Preserves raw reader-factory identity in the Databricks 3.5 and Spark 4.2 scan shims without bypassing runtime exception conversion.

Confidence Score: 5/5

The PR appears safe to merge; the latest factory-wrapper relocation preserves both reader-type identity and runtime missing-file conversion.

No actionable new failure or repository-rule violation remains in the changes since the previous review.

Important Files Changed

Filename	Overview
sql-plugin/src/main/spark350db143/scala/com/nvidia/spark/rapids/shims/GpuBatchScanExec.scala	Preserves the Databricks reader factory's concrete identity and applies missing-file wrapping only when constructing the data-source RDD.
sql-plugin/src/main/spark420/scala/com/nvidia/spark/rapids/shims/GpuBatchScanExec.scala	Preserves the Spark 4.2 reader factory's concrete identity while retaining structured missing-file conversion in the execution path.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuFileNotFoundException.scala	Introduces a path-carrying missing-file exception used across asynchronous reader boundaries.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/io/async/AsyncRunners.scala	Adds ordered non-fatal failure transformation while preserving post-execution hook behavior.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/shims/GpuDataSourceRDD.scala	Converts direct and singly wrapped missing-file failures while preserving metrics cleanup.
sql-plugin/src/main/spark400/scala/com/nvidia/spark/rapids/shims/MissingFileErrorShim.scala	Maps GPU missing-file failures to Spark 4.x structured errors with the exact path parameter.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[GPU batch scan] --> B[Raw partition reader factory]
  B --> C[Reader-type inspection]
  B --> D[MissingFileErrorShim wrapper]
  D --> E[GpuDataSourceRDD]
  E --> F[Columnar reader]
  F -->|missing file| G[Spark-compatible recovery error]

_{Reviews (10): Last reviewed commit: "Preserve multi-file reader factory ident..." | Re-trigger Greptile}

Copilot

Pull request overview

This pull request restores Spark-equivalent missing-file recovery guidance for GPU ORC scans when cached data files are deleted, aligning GPU error messaging with Spark’s V1/V2 expectations and re-enabling previously excluded upstream MetadataCache tests.

Changes:

Add recovery-hint enrichment for FileNotFoundException (including ExecutionException-wrapped cases) in GpuDataSourceRDD, with a switch to include the V1 REFRESH TABLE hint.
Plumb includeRefreshHint = true from V1 file-source scans (GpuFileSourceScanExec) into GpuDataSourceRDD.
Re-enable the previously excluded Spark 3.3 MetadataCache suites and add RAPIDS-focused tests across COALESCING/MULTITHREADED ORC readers.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/utils/RapidsTestSettings.scala	Re-enables the previously excluded MetadataCache tests now that GPU guidance is expected to match Spark.
tests/src/test/spark330/scala/org/apache/spark/sql/rapids/suites/RapidsMetadataCacheSuite.scala	Adds RAPIDS-specific tests asserting V1 vs V2 recovery-hint content across ORC reader modes.
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/GpuFileSourceScanExec.scala	Passes `includeRefreshHint = true` for V1 file-source scans when constructing `GpuDataSourceRDD`.
sql-plugin/src/main/scala/com/nvidia/spark/rapids/shims/GpuDataSourceRDD.scala	Enriches missing-file exceptions with Spark-like recovery guidance, configurable for V1 vs V2 hint text.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

wjxiz1992 · 2026-09-01T03:16:16Z

+          }
      }

      override def next(): Object = {


Performance impact: successful next() calls retain the existing iterator work and metrics-finally path; the added handlers allocate only when a missing-file exception is thrown. Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 · 2026-08-31T10:46:23Z

build

Performance impact: compile-time-only API tightening; runtime policy values and exception-path behavior are unchanged. Signed-off-by: Allen Xu <allxu@nvidia.com>

thirtiseven

LGTM

wjxiz1992 · 2026-09-01T07:42:45Z

build

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 · 2026-09-01T09:47:08Z

build

wjxiz1992 · 2026-09-01T10:00:20Z

build

Signed-off-by: Allen Xu <allxu@nvidia.com>

Resolve the GpuDataSourceRDD conflict by preserving the missing-file conversion while adopting the upstream reader lifecycle and custom metrics integration. Performance impact: the merge adds only exception-path conversion around hasNext/next and one input-partition reference update per reader; the successful row/batch path retains upstream ReaderIterator metric and close behavior. Signed-off-by: Allen Xu <allxu@nvidia.com>

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 · 2026-09-07T04:05:17Z

build

wjxiz1992 · 2026-09-08T02:46:59Z

build

wjxiz1992 · 2026-09-08T02:47:25Z

build

thirtiseven

LGTM

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 · 2026-09-08T04:40:42Z

build

Fix GPU missing-file recovery guidance

07c1035

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 added the bug Something isn't working label Aug 31, 2026

wjxiz1992 requested review from gerashegalov and thirtiseven August 31, 2026 08:45

wjxiz1992 self-assigned this Aug 31, 2026

wjxiz1992 changed the title ~~[WIP] [AutoSparkUT] Fix GPU missing-file recovery guidance~~ [WIP] [AutoSparkUT] Fix GPU missing-file recovery guidance [databricks] Aug 31, 2026

wjxiz1992 changed the title ~~[WIP] [AutoSparkUT] Fix GPU missing-file recovery guidance [databricks]~~ [AutoSparkUT] Fix GPU missing-file recovery guidance [fast-ut] [reduced-it] [databricks] Aug 31, 2026

wjxiz1992 added the test Only impacts tests label Aug 31, 2026

wjxiz1992 marked this pull request as ready for review August 31, 2026 09:15

Copilot AI lite review requested due to automatic review settings August 31, 2026 09:15

Copilot started reviewing on behalf of wjxiz1992 August 31, 2026 09:16 View session

Copilot AI reviewed Aug 31, 2026

View reviewed changes

Handle missing files thrown from datasource next

234f8dc

Performance impact: successful next() calls retain the existing iterator work and metrics-finally path; the added handlers allocate only when a missing-file exception is thrown. Signed-off-by: Allen Xu <allxu@nvidia.com>

thirtiseven reviewed Sep 1, 2026

View reviewed changes

Comment thread sql-plugin/src/main/scala/com/nvidia/spark/rapids/shims/GpuDataSourceRDD.scala Outdated

thirtiseven reviewed Sep 1, 2026

View reviewed changes

Comment thread sql-plugin/src/main/scala/com/nvidia/spark/rapids/shims/GpuDataSourceRDD.scala Outdated

wjxiz1992 changed the title ~~[AutoSparkUT] Fix GPU missing-file recovery guidance [fast-ut] [reduced-it] [databricks]~~ [AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [databricks] [fast-ut] Sep 1, 2026

Require explicit missing-file recovery policy

98650a4

Performance impact: compile-time-only API tightening; runtime policy values and exception-path behavior are unchanged. Signed-off-by: Allen Xu <allxu@nvidia.com>

thirtiseven previously approved these changes Sep 1, 2026

View reviewed changes

wjxiz1992 changed the title ~~[AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [databricks] [fast-ut]~~ [AutoSparkUT] Fix GPU missing-file recovery guidance [fast-ut] [reduced-it] [databricks] Sep 1, 2026

Fix Spark 4 missing-file structured errors

c1e1a06

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 dismissed thirtiseven’s stale review via c1e1a06 September 1, 2026 09:29

Fix AsyncRunners copyright year

ad6d7a5

Signed-off-by: Allen Xu <allxu@nvidia.com>

thirtiseven reviewed Sep 1, 2026

View reviewed changes

Comment thread tests/src/test/spark400/scala/com/nvidia/spark/rapids/MissingFileStructuredErrorSuite.scala Outdated

thirtiseven reviewed Sep 1, 2026

View reviewed changes

Comment thread sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuMultiFileReader.scala Outdated

thirtiseven reviewed Sep 1, 2026

View reviewed changes

Comment thread sql-plugin/src/main/spark400/scala/com/nvidia/spark/rapids/shims/MissingFileErrorShim.scala Outdated

wjxiz1992 changed the title ~~[AutoSparkUT] Fix GPU missing-file recovery guidance [fast-ut] [reduced-it] [databricks]~~ [AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [databricks] [fast-ut] Sep 3, 2026

Merge branch 'main' into codex/fix-15511-orc-missing-file-guidance

844a13f

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 changed the title ~~[AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [databricks] [fast-ut]~~ [AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [fast-ut] [databricks] Sep 3, 2026

wjxiz1992 changed the title ~~[AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [fast-ut] [databricks]~~ [AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [databricks] [fast-ut] Sep 3, 2026

wjxiz1992 added 2 commits September 4, 2026 08:04

Fix GpuDataSourceRDD import order

43d7940

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 changed the title ~~[AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [databricks] [fast-ut]~~ [AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [fast-ut] [databricks] Sep 4, 2026

thirtiseven removed the test Only impacts tests label Sep 7, 2026

wjxiz1992 changed the title ~~[AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [fast-ut] [databricks]~~ [AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [databricks] Sep 7, 2026

Address missing-file review feedback

4896e36

Signed-off-by: Allen Xu <allxu@nvidia.com>

thirtiseven previously approved these changes Sep 8, 2026

View reviewed changes

Preserve multi-file reader factory identity

51b6b89

Signed-off-by: Allen Xu <allxu@nvidia.com>

wjxiz1992 dismissed thirtiseven’s stale review via 51b6b89 September 8, 2026 04:36

wjxiz1992 changed the title ~~[AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] [databricks]~~ [AutoSparkUT] Fix GPU missing-file recovery guidance [reduced-it] Sep 8, 2026

wjxiz1992 requested a review from thirtiseven September 9, 2026 03:28

thirtiseven approved these changes Sep 9, 2026

View reviewed changes

wjxiz1992 merged commit cfdd1dd into NVIDIA:main Sep 9, 2026
59 checks passed

Uh oh!

Conversation

wjxiz1992 commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklists

Uh oh!

greptile-apps Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Flowchart

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

wjxiz1992 Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

wjxiz1992 commented Aug 31, 2026

Uh oh!

Uh oh!

Uh oh!

thirtiseven left a comment

Choose a reason for hiding this comment

Uh oh!

wjxiz1992 commented Sep 1, 2026

Uh oh!

wjxiz1992 commented Sep 1, 2026

Uh oh!

wjxiz1992 commented Sep 1, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wjxiz1992 commented Sep 7, 2026

Uh oh!

wjxiz1992 commented Sep 8, 2026

Uh oh!

wjxiz1992 commented Sep 8, 2026

Uh oh!

thirtiseven left a comment

Choose a reason for hiding this comment

Uh oh!

wjxiz1992 commented Sep 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wjxiz1992 commented Aug 31, 2026 •

edited

Loading

greptile-apps Bot commented Aug 31, 2026 •

edited

Loading